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WATERMARKING AND FINGERPRINTING DIGITAL CONTENT USING 

ALTERNATIVE BLOCKS TO EMBED INFORMATION 



Background of the Invention 

5 

1. Field of the Invention 

The invention relates to distribution of digital content. 

10 2. Related Art 

Distribution of digital content for media streams, such as for example movies, is subject to 
several problems. One problem is that it is easy to make exact copies of digital content, thus 
allowing any recipient of that content to redistribute it, whether or not authorized to do so. It 

15 would be advantageous to be able to distribute digital content, particularly digital content for 
media streams, without fear of its unauthorized distribution. This would be particularly 
advantageous when it is desired to distribute digital content using a communication link, 
such as for example a computer network or other technique for distribution to end viewers 
(for example, either on demand, in anticipation of future demand, or in response to 

20 something else). 

One known solution is to mark the digital content with a "fingerprint," so that an 
unauthorized redistributor of the content can be identified, thus hopefully deterring potential 
unauthorized redistributors. However, fingerprinting of digital content is subject to several 

25 known problems, including (1) that presentation of the media stream might be degraded by 
altering the original content, (2) that recipients might attack the fingerprint, either by altering 
it or erasing it, or (3) that it might be difficult to determine the unauthorized redistributor 
with sufficient certainty, particularly if multiple unauthorized redistributors collude to make 
copies in which the fingerprint has been altered or erased using information available to 

30 those multiple recipients. 

Another problem is ,that the fingerprint might interfere with distribution, decoding, or 
presentation of the original media stream. In particular, when the original media stream is 

1 
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encoded using a known format for movies, such as for example the MPEG-2 format, the 
amount of data included in even a single movie can be quite large. It would be advantageous 
for any fingerprinting method to retain the structure of the original MPEG encoding 
(including packetization of the original media stream), so that the particular fingerprinted 

5 digital content is not changed for distribution purposes by the fingerprint. In particular, it 
would be advantageous to assure that (1) the length of the fingerprinted digital content is the 
same as the length of the original digital content, and (2) sending the fingerprinted digital 
content can be interrupted and restarted, such as when using network communication 
techniques, without having to recompute the fingerprinted digital content and without having 

10 to maintain large amounts of intermediate state information about the fingerprinted digital 
content. 

Summary of the Invention 

15 One aspect of the invention is a method of embedding information in a media stream. The 
method includes (1) selecting a set of locations within that media stream to embed 
information, (2) selecting a set of possible alterations to make at those locations, and (3) 
making a subset of the possible alterations. The set of locations and possible, alterations is 
herein sometimes called a "watermark." The subset of actual alterations that are made is 

20 herein sometimes called a "fingerprint" 

The method determines a set of locations at which there are alternative versions of the same 
media stream, such as one being the original and one being an alternative version of the 
same media stream, herein sometimes called the "original movie" and the "alt-movie" 

25 ("alternative movie"), or sometimes called the "original block" and the "alt-block" 
("alternative block") for a particular block in the media stream. More than one alt-movie or 
alt-block can be defined for each location in order to permit more than one bit of information 
to be embedded at each location. Thus, each such location can embed one or more bits of 
information, the embedded bits being responsive to whether the original movie or the alt- 

30 movie (or one of plural alt-movies) is selected for that location. 

An aspect of the method includes selecting the alterations so that there is no change in 
distribution of the media stream, such as for example (1) packetization of the digital content 

2 
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representing that media stream, or (2) synchronization of audio with video portions of the 
media stream. In a preferred embodiment, unchanged distribution can be accomplished by 
making no changes in lengths of portions of the digital content representing that media 
stream, such as for example individual blocks of an MPEG-encoded movie. 

5 

In a preferred embodiment, the method includes determining a set of intra-coded blocks in 
an MPEG-2 encoding of the digital content for that media stream. Each intra-coded block 
has a set of run-level encoded values for DCT coefficients for that block. The method 
determines if there is an alternative set of values which differ only a little from the original 
10 set of values, and which do not change the bit-length of the encoded block. Each such block 
thus provides both a location to embed information, and an alteration for that location, and 
thus provides at least one bit worth of embeddable information. In alternative embodiments, 
macroblocks may be used instead of blocks. 

[5 This preferred embodiment would include the choice of (1) the number of run-level encoded 
values to be changed in any individual block, such as <= 2 values; (2) the amount of 
alteration, such as 47- 1 level, for any run-level encoded values to be changed; and (3) the 
amount of quantization allowed for any run-level encoded values to be changed, such as a 
quantization scale <= 24. 

10 

In a preferred embodiment, the method includes noting those blocks at which an alt-block 
can be selected, and selecting the particular alt-block at each block in response to a random 
or pseudorandom effect. This type of effect may help a fingerprint resist attacks that attempt 
to study the run-level codes in order to "undo" the fingerprint. It may also help to spread the 

15 watermarking over a relatively large range of spatial frequencies of the content, further 
diminishing visible effects while also helping to resist attacks. Thus, for each movie that is 
watermarked, the particular alt-movie by which it is watermarked is resistant to attacks that 
might involve attempting to reverse the embedding of information by re-encoding alt-blocks 
into original blocks or re-encoding blocks into alt-blocks, because the selection of those alt- 

J0 blocks or those original blocks is not easily known to the attacker. 

The method of the preferred embodiment also excludes blocks at locations that span packet 
boundaries within a packetized data stream for the digital content representing the media 
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stream. Alt-blocks preferably are not selected for such spanning blocks because accessing 
those blocks would involve parsing of the packetized digital content, possibly involving non- 
trivial look-ahead and analysis of the digital content representing the media stream. Thus, 
this optimization simplifies implementation of the preferred embodiment of the invention. 

5 

The invention is not restricted to movies, but is also applicable to other digital content and 
other media streams, such as for example animation or sound, as well as to still media, such 
as for example pictures or illustrations, and to databases and other collections of information. 

10 Brief Description of the Drawings 

Figure 1 shows a block diagram of a system for watermarking and fingerprinting digital 
content. 

15 Figure 2 illustrates MPEG encoding and generation of alt-blocks according to the invention. 

Figure 3 illustrates preservation of packetization according to the invention. 

Figure 4 shows a flow diagram of a method for watermarking and fingerprinting digital 
20 content with a set of locations at which information might be embedded. 

Figure 5 shows a flow diagram of a recursive technique that can be used to identify possible 
alterations in a block for determining alt-blocks according to the invention. 

25 Figure 6 shows a flow diagram of a method for detecting a fingerprint that embeds 
information in digital content. 

Detailed Description of the Preferred Embodiment 

30 In the following description, a preferred embodiment of the invention is described with 
regard to preferred process steps and data structures. Those skilled in the art would 
recognize after perusal of this application that embodiment of the invention can be 
implemented using one or more general-purpose processors or special-purpose processors or 
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other circuits adapted to particular process steps and data structures described herein, and 
that implementation of particular process steps and data structures would not require undue 
experimentation or further invention. 

5 Lexicon 

The following terms refer or relate to aspects of the invention as described below. The 
descriptions of general meanings of these terms are not intended to be limiting, only 
illustrative. 

10 

The phrase "digital content" describes data in a digital format, intended to represent media 
streams or other information for presentation to an end viewer. "Digital content" is 
distinguished from packaging information, such as for example message header information. 

15 The phrase "media stream" describes information intended for presentation in a sequence, 
such as motion pictures including a sequence of frames or fields, or such as audio including 
a sequence of sounds. As used herein, the phrase "media stream" has a broader meaning 
than it's the standard meaning for "streaming media," of sound and pictures that are 
transmitted continuously using packets and that start to play before all of the content arrives. 

20 Rather, as used herein, there is no particular requirement that "media streams" must be 
delivered continuously. Also as described herein, media streams can refer to other 
information for presentation, such as for example animation or sound, as well as to still 
media, such as for example pictures or illustrations, and also to databases and other 
collections of information. 

25 

The phrase "embedding information in media streams" describes generating a set of digital 
content for that media stream, for which the digital content both represents the media stream 
and also includes the embedded information in a form capable of later detection. 

30 The term "watermark" describes a schema for digital content by which information can be 
embedded into that digital content. As described herein, an attacker cannot easily remove 
the watermark provided by the invention. However, the concept of a watermark as described 
herein is sufficiently general to include watermarks that are not so resistant to attack. As 
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described herein, the watermark provided by the invention includes within the media stream, 
both a set of locations at which to embed information, and possible alterations to make at 
those locations by which information is embedded. However, the concept of a watermark as 
described herein is sufficiently general to include watermarks using other techniques for 
5 embedding information. 

The term "fingerprint" describes a specific set of information sufficient to identify at least 
one designated recipient of digital content. As described herein, multiple attackers colluding 
together cannot easily remove the fingerprint provided by the invention, or prevent at least 

10 one of them from being detected as unauthorized distributor of the digital content. However, 
the concept of the fingerprint as described herein is sufficiently general to include 
fingerprints that are not so resistant to removal, or do not provide such capability for 
detecting unauthorized distributors of the digital content. As described herein, the 
fingerprint provided by the invention includes within the media stream, a specific set of 

1 5 alterations to make at the locations identified by the watermark. However, the concept of the 
fingerprint as described herein is sufficiently general to include fingerprints using other 
techniques for embedding information, detecting the embedded information, and detecting 
unauthorized distributors of the digital content. 

20 The term "adaptation" describes a process in which a sender delivers digital content to a 
recipient. As described herein, the sender decrypts its copy of the digital content, embeds, 
information in the media stream represented by that digital content (thus partially 
fingerprinting that digital content), and re-encrypts that partially fingerprinted digital 
content. The sender delivers the adapted digital content to the recipient. 

25 

The phrase "end viewer" describes a recipient of the media stream for whom decoding of the 
digital content for the media stream, and presentation of the media stream, is contemplated. 

The term "decoding" describes generating data in a form for presentation of the media 
30 stream, in response to the digital content for the media stream in an encoded format. As 
described herein, the encoded format might include an industry standard encoded format 
such as MPEG-2. However, the concept of decoding as described herein is sufficiently 
general to include other encoding formats for the media stream. 

6 
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The term "presentation" describes generating information in a form for viewing (or other 
sensing) of the media stream, such as for example audio and visual information for viewing a 
movie. As described herein, presentation of a movie might include visual display of the 
frames or fields of motion picture, as well as audio presentation of a soundtrack associated 
with that motion picture. However, the concept of presentation as described herein is 
sufficiently general to include a wide variety of other forms of generating information for 
reception by end viewers, including audio, visual, or otherwise. 

The phrases "original movie" and "alt-movie" describe alternative versions of the same 
media stream, such as one being an original version of that media stream introduced into a 
system using aspects of the invention, and another being an alternative version of that same 
media stream generated in response to the original movie. Similarly, the phrases "original 
block" and "alt-block" describe alternative versions of the same individual block or 
macroblock within the original movie or alt-movie. As described herein, a difference 
between the original movie and the alt-movie is historical, in that the alt-movie can be 
substituted for the original movie in every respect. Similarly, a difference between any one 
original block and its associated alt-block is historical, in that the alt-block can be substituted 
for the original block in every respect. 

The phrase "MPEG" is an acronym for the "Moving Picture Experts Group," a working 
group of the ISO/IEC (International Organization for Standardization/International 
Engineering Consortium). The phrases "MPEG-1," "MPEG-2," and "MPEG-4" refer to 
techniques for encoding media content as set forth in the ISO/IEC 11172, ISO/IEC 13818, 
and ISO/IEC 14496 documents, respectively. The invention is not limited to use with 
MPEG, and these terms should be read broadly to encompass other encoding techniques, 
whether of similar nature or otherwise. 

The term "intra-coded block" refers to a block that is part of an intra-coded macroblock. 
Intra-coded blocks, and macro-blocks are familiar terms in the art of MPEG techniques. 

Other and further applications of the invention, including extensions of these terms and 
concepts, would be clear to those of ordinary skill in the art after perusing this application. 
These other and further applications are part of the scope and spirit of the invention, and 
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would be clear to those of ordinary skill in the art without further invention or undue 
experimentation. 



System Elements 

5 

Figure 1 shows a block diagram of a system for watermarking and fingerprinting digital 
content. 

System 100 includes video distribution network 110. The video distribution network 
10 includes at least one injection origin 120 and a plurality of end viewer premises 130. 

1 . Distribution from the injection origin to the end viewer premises. 

Injection origin 120 receives digital content 121 for media streams from sources outside the 
15 video distribution network 110. In a preferred embodiment, these sources might include 
content producers or content aggregators, such as for example movie production studios, 
television studios, or radio or television network syndicators. If necessary, injection origin 
120 formats digital content 121, watermarks it, and encrypts it for storage at injection origin 
120. In a preferred embodiment, injection origin 120 uses a method of watermarking and 
20 fingerprinting as described herein. 

In a preferred embodiment, injection origin 120 includes at least a processor capable of 
executing instructions, memory for storing digital content and instructions, and a network 
interface. 

25 

Video distribution network 110 includes network of caching devices 111, each capable of 
acting individually or in concert, to receive, store, and distribute digital content 121 from 
injection origin 120 to end viewer premises 130. In a preferred embodiment, requests from 
end viewers prompt video distribution network 110 to distribute digital content 121 to end 
30 viewer premises 130, thus using a "pull" model for distribution. However, in alternative 
embodiments, injection origin 120 or another device may prompt video distribution network 
110 to distribute digital content 121 to end viewer premises 130, thus alternatively using a 
"push" model or another model for distribution. 
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In a preferred embodiment, caching devices 1 1 1 are disposed in a tiered distribution system, 
including primary caches 112, intermediate caches 113, and leaf caches 114. The primary 
caches 112 receive digital content 121 directly from injection origin 120. Intermediate 
caches 113 receive digital content 121 from primary caches 112, or from other intermediate 
caches 1 13 closer by the network topology to injection origin 120. Leaf caches 1 14 receive 
digital content 121 from intermediate caches 113, or possibly directly from primary caches 
1 12, and distribute digital content 121 directly to end viewer premises 130. 

Preferably (i.e., in a preferred embodiment), each of the caching devices and end viewer 
premises includes at least a processor capable of executing instructions, memory for storing 
digital content and instructions, and a network interface. 

In a preferred embodiment, video distribution network 110 includes a redundant 
communication network, such as the Internet or a secure subnet thereof. However, in the 
context of the invention there is no particular requirement for video distribution network 1 10 
to use any particular communication technique. In alternative embodiments, any 
communication technique capable of delivering copies of digital content 121 from injection 
origin 120, through video distribution network 110, and ultimately to end viewer premises 
130, would also be suitable. 

Distribution of digital content 121 using video distribution network 110 in a preferred 
embodiment provides that copies of digital content 121 might be recorded and maintained at 
multiple caching devices 111, and might be delivered using more than one pathway from 
injection origin 120 to end viewer premises 130. 

For a first example, copies of the digital content 121 might be delivered from the injection 
origin 120 to two different intermediate caches 113 (A and B), and from those intermediate 
caches 113 (A and B) to multiple end viewer premises 130. In the event that one of those 
intermediate caches 113 (A) later discards its copy of the digital content 121, it can receive 
another copy from the other intermediate cache 113 (B) for further delivery to end viewer 
premises 130. 

For a second example, different portions of the digital content 121 might be delivered from 
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the injection origin 120 to different intermediate caches 113 (A and B), and from those 
intermediate caches 1 13 (A and B) to the same end viewer premises 130. This might occur 
if the act of sending the digital content 121 from the injection origin 120 to the end viewer 
premises 130 was interrupted, such as for example by a communication link failure within 
the video distribution network 1 10 or alternatively by user action, and was later resumed and 
completed. 



2. Distribution to each recipient using the video distribution network. 

Whenever any sender delivers digital content 121 to any recipient within the video 
distribution network 110, the sender performs adaptation of the digital content 121 for that 
recipient. In adaptation, the sender decrypts its copy of the digital content, embeds 
information in the media stream represented by that digital content by partially fingerprinting 
that digital content, and re-encrypts that partially fingerprinted digital content. 

In a preferred embodiment, adaptation is performed for all recipients, including both caching 
devices 111 and end viewer premises 130. This would include transfers among caching 
devices 111, even those the same or similar distance from the injection origin 120. 
However, in the context of the invention there is no particular requirement for adaptation to 
be performed for all possible recipients, so that in alternative embodiments, some portion of 
the video distribution network 110 might transmit the digital content 121 through without 
decrypting or re-encrypting it. 

Preferably, a different key is used for decryption and re-encryption. It is also possible to use 
different cryptographic schemes for decryption and re-encryption. Alternatively, the same 
keys, schemes, or both can be used. 

As described above, it is therefore possible as a consequence for individual end viewer 
premises 130 to receive digital content 121 that has been partially fingerprinted for multiple 
recipients. However, in a preferred embodiment, the method of fingerprinting (such as for 
example using a technique for detecting collusion among multiple recipients of fingerprinted 
digital content) provides for detecting individual viewers anyway. 
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When individual end viewer premises 130 receives the encrypted, fingerprinted digital 
content 121, it records that digital content 121 in local video library 13L Local video library 
131 maintains digital content 121 in its encrypted, fingerprinted form for later distribution to 
one or more playback elements 132. In response to a request by an end viewer, local video 
5 library 131 distributes digital content 121 to one or more playback elements 132, at which 
digital content 121 is substantially simultaneously decrypted and presented to end viewers 
for viewing. 

Further detail of one possible implementation of system 100 is discussed in a related 
10 application, "Parallel Distribution and Fingerprinting of Digital Content," (Collens et al.) 
U.S. application number 10/356,692, filed 31 January 2003. 

Methods of Operation 

15 Figure 2 illustrates MPEG encoding and generation of alternative blocks (i.e., alt-blocks) 
according to the invention. The invention is not limited to use with MPEG encoded digital 
content. After reading the following discussion of the invention in the MPEG context, one 
skilled in the art of digital encoding would be able to apply the invention to different 
encoding schemes and data types without undue experimentation or further invention. 

20 

In Figure 2, frame 200 of digital content is processed according to the MPEG standard. For 
illustrative purposes, this frame is shown as an image frame in RGB, CMYK, or some other 
format. The invention is also applicable to other types of digital content. 

25 The data in the frame itself can be processed. If the data in the frame itself is processed, the 
frame is said to be "intra coded." 

I 

Alternatively, the difference of the data from data predicted for the frame can be processed. 
Prediction of frame data can be based on data in a previous frame or on data in both previous 
30 and succeeding frames. In the former case, the frame is said to be a "predictive" frame; in 
the latter case, the frame is said to be a "bidirectional" frame. 



The preferred embodiment of the invention is applied to intra coded frames only; however, 
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Frame 200 is divided into 16x16 pixel macroblocks 210. Each macroblock is then 
decomposed into luminance macroblock 220 and chrominance macroblock 225. Figure 2 
5 illustrates a 4:2:0 chrominance format. In this format, the luminance macroblock contains 
four 8x8 blocks, and the chrominance macroblock contains two 8x8 blocks. Lower 
resolution can be used for the chrominance data because the human eye is less sensitive to 
chrominance variations than to luminance variations. The chrominance blocks can be 
determined through any of various downsampling techniques. 

10 

In other chrominance formats, different size chrominance macroblocks can be used. For 
example, in a 4:2:2 format, each chrominance macroblock contains four 8x8 blocks, and in a 
4:4:4 format, each chrominance macroblock contains eight 8x8 blocks. 

15 Next, each block of luminance and chrominance data is processed. In order to process a 
block, a discrete cosine transform (DCT) is applied to the block, resulting in a set of 8x8 
DCT coefficients 230. 

The DCT coefficients are then quantized. Quantization involves dividing all of the DCT 
20 coefficients for a block by an integer and discarding any remainders. The process results in 
a few non-zero coefficients interspersed among coefficients of zero. The integer by which 
the coefficients are divided is the quantization level for the block. Quantization results in 
some loss of precision; however, depending on the degree of quantization permitted, the loss 
of precision will be acceptable to an end viewer (if noticed at all). The benefit of 
25 quantization is that it allows for a high degree of data compression. 

The quantized data for a block is then ordered by running through the block in a zig-zag 
pattern such as pattern 240. This ordering results in a series of non-zero coefficients, called 
levels, separated by runs of zeros. In Figure 2, such a series is shown representationally at 
30 reference numeral 250. 



The series of levels and zeros are converted into a list of pairs of numbers through a process 
called run level encoding. The pairs of numbers in the list are called run level codes, 
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represented at reference number 260. The first number in each pair represents a number of 
zeros in a run preceding a level, and the second number in each pair represents the level. 
Thus, for example, the series of levels and zeros (A 0 0 0 0 0 B 0 0 ...) can be converted to 
the pairs (0,A) and (5,B), etc., with A and B being values for levels. 

In order to improve compression, the run level codes are then encoded through some 
adaptive coding scheme such as a Huffman code. This type of coding scheme results in 
different length binary codes for each run level code. For this reason, the resulting codes are 
called variable length codes (VLCs). In Figure 2, different length VLCs 270 are represented 
by different length dashes. 

VLCs 270 include four dashes representing four variable length codes, which in turn 
represent four run level codes for quantized DCT coefficients for a single 8x8 pixel block of 
luminance or chrominance data. A block could be represented by fewer or more such VLCs 
of shorter or longer length. 

t 

As discussed above, the conversion of a frame into VLCs involves downsampling and 
quantization; as a result, the VLCs do not represent the exact frame data. The invention 
takes advantage of this fact by determining one or more alternate sets of VLCs for a frame 
that represent the frame in almost, but not exactly, the same way. These alternate sets of 
VLCs form alt-blocks according to the invention. The original set of VLCs is referred to as 
the original block. 

Some or all of the VLCs for the alt-blocks might have different lengths than the VLCs in the 
original block. However, according to the invention, the total length of the VLCs for an alt- 
block should be equal to the total length of the VLCs for the original block. This is shown 
representationally in Figure 2, where the total lengths of the VLCs for alt-blocks 280 are 
equal to the total length of VLCs 270 for original block 290. 

Figure 3 illustrates preservation of packetization according to the invention. 

In Figure 3, data stream 300 includes original blocks 310 of VLCs along with control data 
and the like in headers 320. The variable number and lengths of VLCs in each block are 
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Data stream 300 is packetized for presentation as a media stream as packetized data stream 
330. The packetized data stream includes its own packet headers 340 and packets 350 
including all data from data stream 300. The packet boundaries can occur at any place in 
data stream 300, as shown in Figure 3 by vertical lines between packetized data stream 330 
and data stream 300. Thus, the packet boundaries can occur in blocks 310 and headers 320. 

Data stream 360 includes alt-blocks 370 for some of the blocks in data stream 300. These 
alt-blocks can be inserted into the data stream in place of the corresponding original blocks 
in order to embed data in the data stream. 



As discussed with respect to Figure 2, the total length of the VLCs for an alt-block equals 
the total length of the VLCs for the corresponding original block. However, the number, 
size, or both of the individual VLCs can vary from those in the original blocks, as long as the 
overall block lengths are unchanged. These variations are represented by the dashed lines in 
blocks 370. 



an 



Because the length of an alt-block matches the length of its corresponding original block, 
alt-block can be substituted into packetized data stream 330 by substituting a packet with the 
alt-block for the packet with the original block. 

However, if an alt-block spans two packets, substitution of the alt-block would require non- 
trivial look-ahead and analysis of the packetized data stream. Therefore, the preferred 
embodiment of the invention excludes blocks which span packet boundaries from the 
possible locations for embedding data. In other words, alt-blocks preferably are not 
permitted for blocks that span packet boundaries in the packetized data stream. 

Specific details of a preferred technique for selecting locations and alt-blocks for those 
locations are discussed next. The set of possible locations and alt-blocks for a particular 
item of digital content forms a watermark for that digital content. A copy of the content 
which some or all of those possible alt-blocks have been inserted is said to be wholly 
partially fingerprinted. 



m 
or 
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Figure 4 shows a flow diagram of a method for watermarking and fingerprinting digital 
content with a set of locations at which information might be embedded. 

1 . Watermarking 

A set of block locations and alt-blocks at those locations are a specific example of a set of 
locations and possible alterations in the context of MPEG. In general, a set of such locations 
and possible alterations at those locations forms a watermark according to the invention. 
Step 401 and 402 are the steps for creating this watermark. Preferably, these steps are 
performed at injection origin 120. 

In step 401, locations for embedding information are selected in digital content. In the 
preferred embodiment, this digital content can be an MPEG-1, MPEG-2, or MPEG-4 
(collectively referred to as simply MPEG hereinafter) encoded digital movie. The following 
steps are described in the context of an encoded digital movie. However, the invention is 
equally applicable to other digital content, and application of these steps would not require 
further invention or undue experimentation. 

The locations are preferably 8x8 blocks of either luminance and chrominance DCT 
coefficients, but could also be 16x16 macroblocks of such coefficients. Preferably, the 
locations are blocks of intra coded macroblocks, although the invention can also be used 
with locations that are complete intra coded macroblocks. Blocks and intra coded 
macroblocks are well known in the art of MPEG encoding. 

The invention also can be applied to other blocks that contain coefficients from transforms 
other than DCT, for example wavelets or any other transform in which luminance or 
chrominance information is presented in some transformed domain (e.g., in a non-MPEG 
encoding scheme), or to transforms of RGB data, CMYK data, YUV data, and more 
generally, to any transform in which information useful for presentation of the media stream 
is represented in the digital content. Those skilled in the art, after perusal of this application, 
would recognize that essentially any combination of color space and basis functions would 
be workable, is within the scope and spirit of the invention, and would not require undue 
experimentation or further invention. 
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In the case of packetized digital content, blocks that span packet boundaries preferably are 
excluded as possible locations for embedding information. 

The locations are also selected such that alternative versions of the blocks exist that meet the 
5 criteria discussed in step 402 for selecting possible alterations for the locations. Thus, step 
401 preferably occurs in conjunction with step 402. 

In step 402, possible alterations to the block are selected. In order to maintain packetization 
and synchronization of the digital content, these alterations preferably do not involve any 
10 change to the length of the digital content at the locations (i.e., blocks) where information is 
embedded. 

In the MPEG context, each block has a set of run-level encoded values for discrete cosine 
transform (DCT) coefficients for that block. The invention determines if there is an 
15 alternative set of values which differ only a little from the original set of values, and which 
do not change the bit-length of the encoded block. Each such block thus provides both a 
location to embed information and an alteration for that location, and thus provides at least 
one bit worth of embeddable information. In an alternative embodiment, macroblocks are 
used instead of blocks. 

20 

In more detail, in MPEG, chrominance and luminance blocks are encoded using variable 
length codes (VLCs) corresponding to run level codes that represent discrete cosine 
transforms of pixel data values. These VLCs are Huffinan codes that provide an extremely 
efficient way to compress data. However, the resulting data for different blocks can be 
25 different lengths. 

Thus, in order to maintain constant block length at a location for embedding information, 
any changes to the VLCs for the block must result in a net length difference of zero. This 
can be accomplished in several ways. 

30 

First, one VLC could be substituted for another VLC that has the same length. Second, more 
than one VLC could be substituted for an equal number of VLCs, with the sums of the 
lengths of the original and substituted VLCs being equal. For example, two VLCs with 
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length 5 and 6 could be substituted for VLCs with lengths 4 and 7. Third, one or more VLCs 
could be added or deleted. In order to reduce visible impact, additions and deletions 
preferably are restricted to VLCs with a level of one. Combinations of these three 
techniques, as well as other techniques, can be utilized. 

5 

If more than two VLCs for a block are changed, artifacts can result that might be seen by 
viewers, albeit at possibly a very low level of perception. Therefore, alt-blocks preferably 
are limited to blocks resulting from two changes to VLCs. 

10 In order to limit the visible impact on the presentation of the digital content, changes to the 
data at the identified locations preferably are increases or decreases by a value of one level in 
the run level codes. In addition, MPEG involves quantization of data represented by the run 
level codes. If the quantization scale (QS) is higher than 24, even a level change of one can 
result in unacceptable distortion of the presentation. Thus, only blocks with a quantization 

15 scale of less than or equal to 24 preferably are used to form alt-blocks. 

In sum, in the preferred embodiment, the following criteria are used to identify a possible 
alteration (i.e., alt-block) for an original block: 

20 • the block is part of an intra coded macroblock 

• the block includes one or two run level codes that can be changed by a 
value of plus or minus one level, with the change(s) in run level(s) 
resulting in VLC(s) that have a same length as the VLC(s) for the 

25 unchanged levels (if two run level codes are changed, only the sums of the 

lengths of the VLCs need to be equal) 

• the quantization scale of the block is less than or equal to 24 

30 Experimentally, a typical MPEG-encoded digital movie has been found to include 
approximately one hundred million suitable locations that meet these criteria. 



While use of the foregoing criteria is the preferred embodiment, the invention is not limited 
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to these criteria. Thus, alternative embodiments of the invention can create alt-blocks from 
any types of blocks including blocks contained within macroblocks which are not intra 
coded, can change, add, or delete more than two VLCs in a block, can use changes in levels 
that are greater than one level, and can create alt-blocks from blocks with quantization scales 
greater than 24. Other variations are possible without departing from the scope of the 
invention. 



Figure 5 shows a flow diagram of a recursive technique that can be used to identify possible 
alterations in a block in step 402 for determining alt-blocks. Other different techniques, 
including non-recursive techniques, can be used in performing step 402 without departing 
from the invention. 

In a recursive technique, an initial process is started that in turn calls another instance of 
itself. The called instances of the process can then also call other instances of the process, 
etc. In Figure 5, the recursive call is shown as a dashed line. 

The initial process makes a first permissible modification of a VLC for a block in step 501. 
A modification of a VLC is permissible if it meets the criteria for quantization level and any 
other requirements established for modifications to VLCs. Then, in step 502, the process 
checks to see if the net change in length of the modified VLCs for the block is zero. If the 
net change (including changes from modifications in any calling recursive processes) is zero, 
a possible alt-block has been found. The possible alt-block is added to a list of possible alt- 
blocks in step 503. If the net change is not zero, the change is noted in step 504. 

The process recursively calls another instance of itself in step 507, passing on the net change 
up to that point. The recursively called process starts from the next possible modification of 
a VLC for the block. This next modification might be another modification of the first or 
current VLC or a modification of the next VLC. 

The processes continue to make recursive calls until the end of the block is reached. When 
the end of the block is reached by one of the processes, flow for that process is diverted by 
step 505 to step 506, where the recursive process returns. At this point, the process that 
called the returning process continues at step 508, where a next permissible modification of 
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the VLCs in the block is made. This entire recursive process continues until the initial 
process reaches the end of the block. 

An example is useful for understanding the foregoing recursive process. In this example, 
5 only two changes are tried for each of three VLCs for a block. In order to simplify the 
example, all modifications are assumed to be permissible. These changes are denoted A+l, 
A-l, B+l, B-l, C+l, and C-l. In this example, the following orders of VLC modifications 
are checked for net length changes of zero: (A+l), (A+l, B+l), (A+l, B+l, C+l), (A+l, 
B+l, C-l), (A+l, B-l), (A+l, B-l, C+l), (A+l, B-l, C-l), (A+l, C+l), (A+l, C-l), (A-l), 
10 (A-l, B+l), (A-l, B+l, C+l), (A-l, B+l, C-l), (A-l, B-l), (A-l, B-l, C+l), (A-l, B-l, C-l), 
(A-l, C+l), (A-l, C-l), (B+l), (B+l, C+l), (B+l, C-l), (B-l), (B-l, C+l), (B-l, C-l), 
(C+l), and (C-l). 

The recursive technique of Figure 5 can be used to find all possible alt-blocks for a given 
15 block. However, in alternative embodiments, the technique could be stopped once a certain 
number of possible alt-blocks have been found, or even once a single alt-block has been 
found. If more than one possible alt-block has been found, one or more alt-blocks could be 
selected from the possible alt-blocks, for example pseudorandomly. Other variations are 
possible and remain within the scope of the invention. 

20 

2. Fingerprinting 

Returning to Figure 4, fingerprinting is performed in step 403. Fingerprinting is the process 
of actually making some of the alterations at the selected locations in order to embed 
25 information in the digital content. In the MPEG context, alterations are made by using alt- 
blocks in place of some original blocks. 

If one alt-block is found for a block, that alt-block can be used to embed one bit at that 
location. For example, once the set of locations and possible alterations are defined, a "0" 
30 can be embedded at a location by using an original block at that location, and a "1" can be 
embedded by using an alt-block at the location. Other schemes for embedding the data also 
can be used. 
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Furthermore, if more than one alt-block is found for a block at a selected location, those alt- 
blocks can be used to embed more than one bit at that location. For example, the original 
block could represent "00," a first alt-block could represent "01," a second alt-block could 
represent "10," and a third alt-block could represent "11." Again, other schemes for using 
multiple alt-blocks are possible. 

In step 403, information (i.e., bits) are embedded in the digital content by substituting alt- 
blocks for original blocks at some of the selected locations in the digital content. Preferably, 
only a subset of the locations are subject to change at each of plural different points in 
distribution of the digital content. For example, a first subset can be subject to change at 
primary caches 1 12, a second subset can be subject to change at intermediate caches 1 13, a 
third subset can be subject to change at leaf caches 114, etc. Typically, not all of the 
locations in each subset will be altered because at least some "0s" (or other values depending 
on the coding scheme) will be embedded at the available locations. 

3. Extracting Fingerprint Information 

Fingerprinting is only meaningful if the embedded information can be extracted. Figure 6 
shows a flow diagram of a method for extracting embedded information from a fingerprint in 
digital content. The data can be extracted at computing device that can perform the steps 
shown in Figure 6. Such a computing device preferably includes at least a processor and a 
memory for storing the digital content and for storing instructions for performing the steps. 

In step 601, a watermark for a specific item of digital content is determined. This watermark 
includes a set of locations and possible alterations (e.g., alt-blocks) at those locations for the 
specific item of digital content. 

The watermark itself preferably is not included with digital content that is distributed to end 
users. Therefore, the watermark preferably is retrieved from injection point 120 based on 
identifying information about the digital content. Examples of such identifying information 
include the content's title, revision number, serial number, and the like. 

The identifying information preferably is itself embedded in the digital content in some 
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robust fashion or is self-evident from the digital content. For example, in the case of an 
MPEG encoded movie, the identifying information could be the title of the movie, which 
would be self-evident from the content of the movie. Other identifying information can be 
utilized. 

Once the watermark is retrieved, the selected locations for embedding information defined 
by the watermark can be examined in step 602. The locations are examined so as to 
determine which of the locations have been altered in accordance with the possible 
alterations specified by the watermark. 

Next, in step 603, the embedded information can be extracted from an examination of which 
alterations (e.g., alt-blocks) are or are not present. For example, alterations that are present 
can represent digital "Is," and alterations that are not present can represent digital "0s." 
Other encoding schemes can be used without departing from the invention. 

Preferably, each embedded bit of information is embedded many times in the digital content. 
Thus, loss or corruption of the alterations at some of the locations will not result in loss of 
embedded information. 

In addition, the order of the locations in the digital content preferably is related to the order 
of bits in the embedded information in some known but random (or pseudorandom) fashion. 
This scrambling of the embedded information helps prevent unauthorized access to or 
corruption of the embedded information. 

The digital content that is being examined might have been altered, either intentionally or 
unintentionally. As a result, any given block in the digital content might not exactly match 
either an original block or an alt-block from the content's watermark. In order to address this 
issue, the distance of each block from the corresponding original block in some vector space, 
for example a vector space of DCT coefficients derived from VLCs, could be calculated. 
Then, blocks that are within some distance of an original block or an alt-block could be 
deemed to match those blocks. If each bit of embedded data is embedded multiple times, 
enough bits are likely to be extracted in order to calculate the embedded data. 



21 



WO 2004/006168 
Generality of the Invention 



PCTYUS2003/021592 



The generation of alt-blocks is not dependent upon the type of encoding used for the digital 
content. In a general sense, the watermarking and fingerprinting techniques of the invention 
5 can be applied to any digital data that models a continuous or analog physical process. The 
invention can be applied to digitized audio data, measurement data, video data, other 
multimedia data, and the like, whether encoded by MPEG, some variation of MPEG that 
may or may not use Huffman coding, or some other entirely different encoding scheme. 
With any of this digitized data, the techniques of the invention can be used to select locations 
10 in the digital content and to select possible alterations to make at the selected locations so as 
to embed data, all while preserving packetization of the data. 

Furthermore, the invention is useful for, and has sufficient generality for, applications other 
than distribution of media streams, and other than distribution of digital content. For 
15 example, the invention is also generally useful for applications in which security of datasets 
or identifying recipients of those datasets is desired. 



Thus, although preferred embodiments are disclosed herein, many variations are possible 
which remain within the concept, scope, and spirit of the invention. These variations would 
20 become clear to those skilled in the art after perusal of this application. 
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1 . A method of marking digital content, including steps of 
selecting locations in the digital content for embedding data; 

5 selecting possible alterations to make at the selected locations, with the 

possible alterations preserving packetization of the digital content, and with each possible 
alteration embedding one or more bits of the data in the digital content; 

wherein the locations and the possible alterations together form a watermark 
for the digital content. 

10 

2. A method as in claim 1, wherein the step of selecting locations includes 
steps of identifying blocks in which data can be embedded. 

3. A method as in claim 2, wherein the step of selecting possible alterations 
15 includes steps of identifying one or more alterations that, when applied to a block, do not 

result in a changed block length for the block. 

4. A method as in claim 3, wherein the digital content is encoded using run 
level codes, and wherein the possible alterations for each location use same length run level 

20 codes. 

5. A method as in claim 3, wherein the possible alterations for a block do not 
change the presentation of the block by more than a predefined amount. 

25 6. A method as in claim 5, wherein the predefined amount is defined in terms 

of a maximum level change and a maximum quantization scale. 

7. A method as in claim 6, wherein the maximum level change is plus or 
minus one and the maximum quantization scale is twenty-four. 

30 

8. A method as in claim 1, including the steps of actually making some of the 
possible alterations at the selected locations, with the actual alterations forming a complete 
or partial fingerprint for the digital content. 
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9. A method as in claim 8, wherein portions of the actual alterations are made 
at plural different client, server, or cache devices. 

10. A method as in claim 1, wherein the digital content is an MPEG encoded 

5 movie. 

1 1. A method of detecting a fingerprint in digital content, including steps of 
determining a watermark for the digital content, the watermark including 

locations and possible alterations at those locations for embedding data; 
1 0 determining which of the possible alterations have been made at the locations; 

and 

extracting bits in response to the step of determining which of the possible 
alterations have been made. 

15 12. Digital content, including 

data in a digital format; and 

a watermark including possible alterations to the data at selected locations, 
with the possible alterations preserving packetization of the digital content, and with each 
possible alteration embedding one or more bits of the data in the digital content. 

20 

13. Digital content as in claim 12, wherein the selected locations are blocks 
in which data can be embedded. 

14. Digital content as in claim 13, wherein each of the possible alterations, 
25 when applied to a block, does not result in a changed block length for the block. 

15. Digital content as in claim 14, wherein the digital content is encoded 
using run level codes, and wherein the possible alterations for each location use same length 
run level codes. 



30 



16. Digital content as in claim 14, wherein the possible alterations for a block 
do not change the presentation of the block by more than a predefined amount. 
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17. Digital content as in claim 16, wherein the predefined amount is defined 
in terms of a maximum level change and a maximum quantization scale. 



18. Digital content as in claim 17, wherein the maximum level change is plus 
5 or minus one, and the maximum quantization scale is twenty-four. 

19. Digital content as in claim 12, including a complete or partial fingerprint 
including actual alterations made at some or all of the selection locations. 

10 20. Digital content as in claim 19, wherein portions of the actual alterations 

were made at plural different client, server, or cache devices. 

, 21. Digital content as in claim 12, wherein the digital content is an MPEG 
encoded movie. 

15 

22. A device capable of watermarking digital content, including 

a network interface over which the digital content can be injected into a 

network; 

a processor; and 

20 a memory that stores the digital content and that stores instructions executable 

by the processor to create a watermark for the digital content, the instructions including the 
steps of (a) selecting locations in the digital content for embedding data, and (b) selecting 
possible alterations to make at the selected locations, with the possible alterations preserving 
packetization of the digital content, and with each possible alteration embedding one or more 

25 bits of the data in the digital content, wherein the locations and the possible alterations 
together form the watermark for the digital content. 

23. A device as in claim 22, wherein the device is an injection point for a 
distribution network for the digital content. 

30 

24. A device as in claim 22, wherein the step of selecting locations includes 
identifying blocks in which data can be embedded. 
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25. A device as in claim 22, wherein the step of selecting possible alterations 
includes the step of identifying one or more alterations that, when applied to a block, do not 
result in a changed block length for the block. 

26. A device as in claim 25, wherein the digital content is encoded using run 
level codes, and wherein the possible alterations for each location use same length run level 

codes. 

27. A device as in claim 25, wherein the possible alterations for a block do 
not change the presentation of the block by more than a predefined amount. 

28. A device as in claim 27, wherein the predefined amount is defined in 
terms of a maximum level change and a maximum quantization scale. 

29. A device as in claim 28, wherein the maximum level change is plus or 
minus one and the maximum quantization scale is twenty-four. 

30. A device as in claim 22, wherein the instructions include the step of 
actually making some of the possible alterations at the selected locations, with the actual 
alterations forming a complete or partial fingerprint for the digital content. 

3 1 . A device as in claim 22, wherein the digital content is an MPEG encoded 

movie. 

32. A device capable of partially or fully fingerprinting digital content that 
has been watermarked, with the watermark including locations and possible alterations at 
those locations in the digital content, the device including 

a network interface over which the digital content can be sent to or received 
from a network; 

a processor; and 

a memory that stores the digital content and that stores instructions executable 
by the processor to fingerprint the digital content, the instructions including the step of 
actually making some of the possible alterations at the locations. 
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33. A device as in claim 32, wherein the device is an injection point, caching 
device, or end viewer premises in a distribution network for the digital content. 



34. A device capable of detecting a fingerprint in digital content, including 
5 a processor; and 

a memory that stores the digital content and that stores instruction executable 
by the processor to detect the fingerprint, the instructions including the steps of (a) 
determining a watermark for the digital content, the watermark including locations and 
possible alterations at those locations for embedding data, (b) determining which of the 
10 possible alterations have been made at the locations, and (c) extracting bits in response to the 
step of determining which of the possible alterations have been made. 

35. A memory storing information including instructions, the instructions 
executable by a processor to watermark digital content, the instructions including the steps of 

15 selecting locations in the digital content for embedding data; 

selecting possible alterations to make at the selected locations, with the 
possible alterations preserving packetization of the digital content, and with each possible 
alteration embedding one or more bits of the data in the digital content; 

wherein the locations and the possible alterations together form the 
20 watermark for the digital content. 

36. A memory as in claim 35, wherein the device is an injection point for a 
distribution network for the digital content. 

25 37. A memory as in claim 35, wherein the step of selecting locations includes 

identifying blocks in which data can be embedded. 

38. A memory as in claim 35, wherein the step of selecting possible 
alterations includes the step of identifying one or more alterations that, when applied to a 

30 block, do not result in a changed block length for the block. 

39. A memory as in claim 38, wherein the digital content is encoded using 
run level codes, and wherein the possible alterations for each location use same length run 
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40. A memory as in claim 38, wherein the possible alterations for a block do 
not change the presentation of the block by more than a predefined amount. 

5 

41 . A memory as in claim 40, wherein the predefined amount is defined in 
terms of a maximum level change and a maximum quantization scale. 

42. A memory as in claim 41, wherein the maximum level change is plus or 
0 minus one and the maximum quantization scale is twenty-four. 

43. A memory as in claim 35, wherein the instructions include the step of 
actually making some of the possible alterations at the selected locations, with the actual 
alterations forming a complete or partial fingerprint for the digital content. 

.5 

44. A memory as in claim 35, wherein the digital content is an MPEG 
encoded movie. 

45. A memory storing information including instructions, the instructions 
10 executable by a processor to partially or fully fingerprint digital content that has been 

watermarked, with the watermark including locations and possible alterations at those 
locations in the digital content, the instruction including the step of actually making some of 
the possible alterations at the locations. 

>5 46. A memory as in claim 45, wherein the device is an injection point, 

caching device, or end viewer premises in a distribution network for the digital content. 

47. A memory storing information including instructions, the instructions 
executable by a processor to detect a fingerprint in digital content, the instructions including 
30 the steps of: 

determining a watermark for the digital content, the watermark including 
locations and possible alterations at those locations for embedding data; 

determining which of the possible alterations have been made at the locations; 
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and 

extracting bits in response to the step of determining which of the possible 
alterations have been made. 
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